Cut DB load: drop jobs realtime, remove dead index by simonsmallchua · Pull Request #395 · Good-Native/hover

simonsmallchua · 2026-05-27T13:11:07Z

Summary

Two low-risk database-load reductions, isolated from worker logic:

Remove public.jobs from the supabase_realtime publication. Decoding the WAL for every job-counter tick (running_tasks bumps, recalculate_job_stats, the update_job_queue_counters trigger) was ~14% of total DB time via realtime.list_changes — billed continuously regardless of how many dashboards were subscribed. The frontend already polls the lane-isolated /v1/jobs API as a fallback, so this trades a constant tenant-wide cost for cheap, viewer-scoped polling. notifications stays on the publication.
Convert the job-list frontend to adaptive poll-only. subscribeToJobUpdates now polls (500 ms while a job is active, 1 s idle, re-evaluated each tick) instead of opening a now-dead Realtime channel. Both consuming paths already ran this way on staging where Realtime is unavailable. Removed the now-unused realtime constants and two dead imports.
Drop the redundant idx_domain_hosts_domain_id index (0 scans; the composite (domain_id, host) unique index already serves every domain_id-prefix lookup). Pure write overhead removed from every domain_hosts upsert.

Why

Under heavy job load the worker's adaptive DB-pressure controller pinned at its concurrency floor (HOVER-JD) because query latency stayed above threshold. The Realtime WAL decode was a large, constant contributor to DB CPU contention; removing it lowers latency across the board. The hottest query (promote_waiting_with_outbox, ~58%) is intentionally left untouched — its per-call SQL is already optimal and the only lever is a riskier call-volume change, deferred pending re-measurement after this lands.

Risk

Low. The publication change is reversible (ALTER PUBLICATION supabase_realtime ADD TABLE public.jobs). The polling behaviour is already exercised on staging/preview. API polls hit the raw connection pool, not the pressure-controlled crawl lane, with 1–2 ms indexed queries.

Test plan

Apply migrations to a branch DB; confirm jobs no longer in pg_publication_tables and idx_domain_hosts_domain_id is gone
Load dashboard + Webflow job list; confirm job rows update via /v1/jobs polling (500 ms active / 1 s idle) with no console errors
Re-run load test; confirm HOVER-JD frequency and exec_ema_ms drop, and realtime.list_changes falls out of the top pg_stat_statements entries

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}

Summary by CodeRabbit

Chores
- Dropped a redundant database index and removed job rows from the realtime publication to reduce write and replication overhead.
Refactor
- Job status updates now use adaptive polling (faster when jobs are active, slower when idle) instead of realtime subscriptions; polling cleans up timers on stop.
Documentation
- CHANGELOG updated to record the polling switch and the index/publication removals.

supabase · 2026-05-27T13:11:21Z

Updates to Preview Branch (perf/reduce-db-load) ↗︎

Deployments	Status	Updated
Database	✅	Wed, 27 May 2026 13:23:19 UTC
Services	✅	Wed, 27 May 2026 13:23:19 UTC
APIs	✅	Wed, 27 May 2026 13:23:19 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks	Status	Updated
Configurations	✅	Wed, 27 May 2026 13:23:21 UTC
Migrations	✅	Wed, 27 May 2026 13:23:22 UTC
Seeding	✅	Wed, 27 May 2026 13:23:24 UTC
Edge Functions	✅	Wed, 27 May 2026 13:23:28 UTC

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

coderabbitai · 2026-05-27T13:11:27Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 131ff920-7968-435e-8742-ceb73a5a11f9

📥 Commits

Reviewing files that changed from the base of the PR and between 875ae4b and 4ed405d.

📒 Files selected for processing (1)

web/static/app/pages/webflow-jobs.js

🚧 Files skipped from review as they are similar to previous changes (1)

web/static/app/pages/webflow-jobs.js

📝 Walkthrough

Walkthrough

This PR removes public.jobs from Supabase Realtime, drops a redundant domain_hosts index, and replaces client realtime subscriptions with an adaptive polling loop that toggles between active and idle intervals.

Changes

Realtime Job Updates to Adaptive Polling

Layer / File(s)	Summary
Database index optimization `supabase/migrations/20260527224118_drop_dead_domain_hosts_index.sql`	Drops the redundant standalone `(domain_id)` index on `domain_hosts`, keeping the composite unique index for domain-prefixed lookups.
Disable realtime publication for jobs `supabase/migrations/20260527224119_disable_jobs_realtime.sql`	Conditionally removes `public.jobs` from the `supabase_realtime` publication using an idempotent guard that checks `pg_publication_tables` before altering.
Adaptive polling implementation `web/static/app/pages/webflow-jobs.js`, `web/static/app/pages/dashboard.js`, `CHANGELOG.md`	Replaces realtime subscription with an adaptive polling loop (`POLLING_INTERVAL_ACTIVE_MS` / `POLLING_INTERVAL_IDLE_MS`) that re-evaluates `window.dataBinder?.hasRealtimeActiveJobs` each tick; updates dashboard comments and changelog entries.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Good-Native/hover#208: Both PRs modify the jobs realtime/subscription mechanism; this PR removes the publication and converts the client to polling.

Poem

🐇 I swapped the stream for steady ticks of time,
Polls hum gently—active then idle rhyme,
An index pruned, the publication cleared,
Simpler loops now keep the jobs endeared,
Tiny hops of 500ms make things fine.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main changes: removing jobs from realtime publication and dropping a redundant database index to reduce DB load.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch perf/reduce-db-load

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@supabase/migrations/20260527224118_drop_dead_domain_hosts_index.sql`:
- Line 14: The migration contains a destructive step "DROP INDEX IF EXISTS
public.idx_domain_hosts_domain_id" which violates the additive-only policy;
instead remove the DROP INDEX statement from this migration and either (a)
convert it to a non-destructive, additive change (e.g., create a replacement
index or a no-op migration that records intent), or (b) move the DROP of
idx_domain_hosts_domain_id into an operational/manual maintenance playbook and
leave an in-repo comment or follow-up migration that gates the destructive step
after observation; update the migration to preserve only additive operations and
reference the index name idx_domain_hosts_domain_id and the DROP INDEX action in
the associated maintenance note.

In `@web/static/app/pages/webflow-jobs.js`:
- Around line 177-183: The scheduler currently fire-and-forgets onUpdate()
causing overlapping polls and unhandled rejections; change the setTimeout
callback in schedule() to an async function, await onUpdate() inside a try/catch
to handle errors, and call schedule() from a finally block so the next timer is
always scheduled (referencing schedule(), onUpdate(), timer, stopped, and
intervalMs()).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 40091654-7407-48ec-8943-0f4f478283b7

📥 Commits

Reviewing files that changed from the base of the PR and between 463cced and 8a4e36e.

📒 Files selected for processing (4)

supabase/migrations/20260527224118_drop_dead_domain_hosts_index.sql
supabase/migrations/20260527224119_disable_jobs_realtime.sql
web/static/app/pages/dashboard.js
web/static/app/pages/webflow-jobs.js

github-actions · 2026-05-27T13:16:06Z

Release Versions

App patch: v0.34.19 → v0.34.20

Changelog

Changed

Rate-limited (429/403/503) tasks now retry up to 8 times before hard-failing,
raised from 3, so a domain that throttles hard is paced-and-deferred rather
than failed mid-job. Tunable via GNH_RATE_LIMIT_MAX_RETRIES (previously
documented but never wired).
Every job now starts paced instead of bursting the full per-job concurrency at
a domain. Previously only a never-crawled domain got a warm-up delay; a domain
with a learned (or zero) adaptive delay seeded un-paced and could trigger an
instant 429 storm. Job starts are now floored to a minimum seed delay, so a
previously-crawled domain ramps up from a few concurrent requests and widens
on success. Tunable via GNH_PACER_START_FLOOR_DELAY_MS (0 restores the
prior behaviour).
Dashboard and Webflow job lists now refresh via adaptive polling (500 ms while
a job is active, 1 s when idle) instead of Supabase Realtime. public.jobs
was removed from the supabase_realtime publication because decoding its WAL
on every job-counter change was ~14% of total database time, billed
continuously regardless of how many dashboards were subscribed.
notifications realtime is unchanged. Reverse with
ALTER PUBLICATION supabase_realtime ADD TABLE public.jobs;.

Removed

Redundant idx_domain_hosts_domain_id index (0 recorded scans; the composite
(domain_id, host) unique index already serves every domain_id-prefix
lookup), removing write overhead on every domain_hosts upsert.

codecov · 2026-05-27T13:17:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

github-actions · 2026-05-27T13:26:28Z

🐝 Review App Deployed

Homepage: https://hover-pr-395.fly.dev
Dashboard: https://hover-pr-395.fly.dev/dashboard

Cut DB load: drop jobs realtime, remove dead index

Drop jobs realtime, remove dead index

8a4e36e

Add changelog entry

875ae4b

coderabbitai Bot reviewed May 27, 2026

View reviewed changes

Comment thread supabase/migrations/20260527224118_drop_dead_domain_hosts_index.sql

Comment thread web/static/app/pages/webflow-jobs.js

Await poll update to prevent overlap

4ed405d

simonsmallchua merged commit 466499e into main May 27, 2026
21 checks passed

simonsmallchua deleted the perf/reduce-db-load branch May 27, 2026 14:01

simonsmallchua added a commit that referenced this pull request May 27, 2026

Merge pull request #395 from Good-Native/perf/reduce-db-load

bea4bb1

Cut DB load: drop jobs realtime, remove dead index

simonsmallchua mentioned this pull request May 27, 2026

Reduce promote_waiting_with_outbox DB cost (~58% of DB time) #396

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cut DB load: drop jobs realtime, remove dead index#395

Cut DB load: drop jobs realtime, remove dead index#395
simonsmallchua merged 3 commits into
mainfrom
perf/reduce-db-load

simonsmallchua commented May 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

supabase Bot commented May 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 27, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 27, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 27, 2026

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

simonsmallchua commented May 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Risk

Test plan

Summary by CodeRabbit

Uh oh!

supabase Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Versions

Changelog

Changed

Removed

Uh oh!

codecov Bot commented May 27, 2026

Codecov Report

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

simonsmallchua commented May 27, 2026 •

edited by coderabbitai Bot

Loading

supabase Bot commented May 27, 2026 •

edited

Loading

coderabbitai Bot commented May 27, 2026 •

edited

Loading

github-actions Bot commented May 27, 2026 •

edited

Loading